test(e2e): add reusable QEMU infrastructure for E2E tests by drew · Pull Request #2471 · NVIDIA/OpenShell

drew · 2026-07-24T17:55:59Z

Summary

Add reusable Nix and QEMU infrastructure for running OpenShell tests in disposable Linux VMs. The harness provides pinned native-architecture cloud guests, composable runtime and security configurations, artifact installation, predictable cleanup, and retained diagnostics for failures.

This is the base infrastructure PR for the VM-backed release and E2E test integrations in the child PRs below.

Stack

This is the parent PR.

ci(release): run Linux release canaries in QEMU VMs #2472 — run Linux release canaries in QEMU VMs
test(e2e): add VM-backed E2E suite runner #2473 — run E2E suites through the QEMU VM infrastructure

Related Issue

feat: E2E and release testing improvements #2527
Prototype alternative to feat(e2e): use xtask for host Linux workflows #2254.

Changes

Add pinned Ubuntu 24.04, Fedora 44, CentOS Stream 10, and Rocky Linux 9 cloud-image definitions for ARM64 and AMD64
Add a nix run .#test-vm interface with disposable QCOW2 overlays, cloud-init SSH bootstrap, hardware acceleration where available, cleanup, and retained debug state
Add composable Ansible configurations for Docker, rootless Podman, and SELinux enforcing mode
Support repeatable package installation and direct executable copying into guests
Add automatic KVM-to-TCG fallback for Linux hosts without accessible hardware virtualization
Add dpkg to the Nix development shell and keep Debian package creation artifact-only
Document requirements, supported combinations, package workflows, lifecycle, and current limitations

Testing

Checklist

Follows Conventional Commits
Commits are signed off (DCO)
Architecture and VM usage documentation updated

copy-pr-bot · 2026-07-24T17:56:02Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

copy-pr-bot · 2026-07-28T18:54:28Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew

gator-agent

PR Review Status

Validation: This maintainer-authored PR is project-valid as reusable QEMU/Nix test infrastructure for the release and E2E stack.
Head SHA: 52cb5e1b90ababfdf89a4ee95a37bb111509263f

Review findings:

Warning: the OCI cache path pulls and boots a full guest disk from a mutable repository reference. Metadata and disk checksums protect transfer integrity, but not provenance. Before wiring this into trusted CI, require an org-controlled immutable registry policy or artifact signing/verification; otherwise the cache repository becomes a supply-chain execution surface.
Three line-specific warnings are inline.

Docs: Fern docs/navigation are not required for this contributor/test infrastructure change; nix/test-guest/README.md and architecture/build.md cover the new workflow.

Next state: gator:in-review

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew

gator-agent

PR Review Status

Validation: This maintainer-authored PR remains project-valid as reusable Nix/QEMU test guest infrastructure for release and E2E follow-on work.
Head SHA: 49ae902c28f3c514535434e65bc4dc2c640be5da

Review findings:

Warning: package verification coverage regressed when the Debian package smoke-test block was removed before an equivalent VM-backed CI replacement exists.
Warning: OCI prepared-disk cache pulls still verify integrity but not publisher provenance; trusted CI use needs an immutable org-controlled reference or signature/manifest verification.
Suggestion: nix/test-guest/run.sh should use the same explicit option-value validation pattern as cache.sh for a cleaner CLI failure mode.

Docs: Fern docs/navigation are not required for this contributor/test infrastructure change; nix/test-guest/README.md and architecture/build.md cover the workflow.

Checks: DCO and gate-status publication passed; Branch Checks and Helm Lint are still pending on /ok to test mirror validation. I am not triggering CI while review warnings remain unresolved.

Next state: gator:in-review

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew · 2026-07-28T20:46:43Z

/ok to test 9f24004

drew

gator-agent

PR Review Status

Validation: This maintainer-authored PR remains project-valid as reusable Nix/QEMU test guest infrastructure for release and E2E follow-on work.
Head SHA: 9f240044655387561c88b4b305e8770f88042765

Review findings:

Warning: the SSH control socket path can exceed Unix socket path limits during cache builds because TMPDIR may be a deep cache staging path.
Warning: local prepared-disk cache hits still trust existing metadata without revalidating the disk checksum or QCOW2 structure.

Docs: Fern docs/navigation are not required for this contributor/test infrastructure change; nix/test-guest/README.md and architecture/build.md cover the workflow.

Checks: DCO and gate-status publication passed; Branch Checks and Helm Lint are still pending on /ok to test mirror validation. I am not triggering CI while review warnings remain unresolved.

Next state: gator:in-review

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew

gator-agent

PR Review Status

Validation: This maintainer-authored PR remains project-valid as reusable Nix/QEMU test guest infrastructure for release and E2E follow-on work.
Head SHA: 7f365efb813777575f291f6e48230913aa287303

Review findings:

Warning: removing the Debian package smoke block drops existing package verification coverage before the VM-backed replacement lands.
Suggestion: the runner option parser should validate required option values before shift 2 for cleaner CLI failures.

Docs: Fern docs/navigation are not required for this contributor/test infrastructure change; nix/test-guest/README.md and architecture/build.md cover the workflow.

Checks: DCO and gate-status publication passed; Branch Checks and Helm Lint are still pending on /ok to test mirror validation. I am not moving to pipeline watch while the review warning remains unresolved.

Next state: gator:in-review

drew

gator-agent

PR Review Status

Validation: This maintainer-authored PR remains project-valid as reusable Nix/QEMU test guest infrastructure for release and E2E follow-on work.
Head SHA: 418b2605f9404612b71674c2fd225d2312f5a879

Review findings:

Warning: the new README documents E2E wrapper paths and a mise task that are not present in this PR/current tree.
Warning: Debian package smoke verification is removed before equivalent VM-backed coverage lands.
Warning: the OCI cache push path accepts an existing remote tag without validating the artifact contents against the local cache key.
Warning: SSH/SCP should force the generated key with IdentitiesOnly=yes to avoid agent-key authentication failures.
Suggestion: align the runner option parser with cache.sh value validation for cleaner CLI failures.

Docs: Fern docs/navigation are not required for this internal contributor/test infrastructure change; nix/test-guest/README.md and architecture/build.md cover the intended workflow, but the README includes stale future entry points that need correction.

Checks: DCO and gate-status publication passed; Branch Checks and Helm Lint remain pending on /ok to test mirror validation. I am not moving to pipeline watch while review findings remain unresolved.

Next state: gator:in-review

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew

gator-agent

PR Review Status

Validation: This maintainer-authored PR remains project-valid as reusable Nix/QEMU test guest infrastructure for release and E2E follow-on work.
Head SHA: 86cf640591154b8934db8e85483f20dd3c623c37

Review findings:

No blocking findings remain. The independent reviewer re-checked the current head and found no line-specific actionable findings.

Trusted author/maintainer feedback: Thanks @drew, I checked your current-head replies about the prior cache validation, OCI tag verification, SSH identity, parser validation, stale README, and Debian package-smoke concerns. The code changes address the actionable review items, and the Debian package-smoke removal is covered by the release-canary path you called out.

Docs: Fern docs/navigation are not required for this internal contributor/test infrastructure change; nix/test-guest/README.md and architecture/build.md cover the workflow.

Tests: Applying test:e2e because this adds VM-backed runtime/test infrastructure and package execution paths.

Checks: DCO and gate-status publication are passing. Branch Checks and Helm Lint are pending current-head /ok to test mirror validation.

Next state: gator:watch-pipeline

drew · 2026-07-28T23:27:05Z

/ok to test 86cf640

github-actions · 2026-07-28T23:27:11Z

Label test:e2e applied for 86cf640. Open Branch E2E Checks, find the run for commit 86cf640, and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew

gator-agent

PR Review Status

Validation: This maintainer-authored PR remains project-valid as reusable Nix/QEMU test infrastructure for release and E2E work.
Head SHA: 3d48a73ab33a8f9554e13d8b320bfe79088d17ea

Review findings:

Three line-specific warnings are inline: host-side scp option injection, Podman relay readiness, and mutable runtime-image inputs.
Suggestion: add focused mocked shell tests for hostile argument shapes and cache metadata/checksum/digest rejection paths; keep full QEMU integration coverage in the stacked child PRs.

Durable feedback: Prior trusted maintainer resolutions and waivers remain honored. These findings are new or materially distinct: the scp concern is host-side option parsing rather than the resolved guest-destination quoting issue, and the two Podman findings arise from this head's new hook.

Docs: Fern docs/navigation are not required for this contributor/test infrastructure; nix/test-guest/README.md and architecture/build.md cover the workflow.

Checks: DCO and required-gate publication pass. Branch Checks, Helm Lint, and E2E remain pending, but author changes are needed before pipeline watch resumes.

Next state: gator:in-review

drew · 2026-07-29T04:29:30Z

+	fi
+	remote_package="/home/openshell/${package_name}"
+	echo "==> Installing package: ${package_name}"
+	scp "${scp_args[@]}" "${package}" "openshell@127.0.0.1:${remote_package}"


gator-agent

Warning: User-controlled --install paths are passed directly to scp; line 563 does the same for --copy. A valid path beginning with an option such as -S... can alter host-side scp behavior (CWE-88). Resolve validated sources to absolute paths with realpath -- before basename/scp, use an end-of-options marker where supported, and add a leading-hyphen regression case.

drew · 2026-07-29T04:29:30Z

+              TCP-LISTEN:8080,bind=0.0.0.0,reuseaddr,fork \
+              "UNIX-CONNECT:${relay_socket}"
+          for _ in $(seq 1 50); do
+              if systemctl --user is-active --quiet "${network_relay_unit}.service"; then


gator-agent

Warning: systemctl is-active only proves that systemd launched this default Type=simple unit; it does not prove socat has bound port 8080 inside the rootless network namespace. Poll the actual listener from podman unshare --rootless-netns while also checking that the unit remains active, so the hook cannot return before the relay is ready.

drew · 2026-07-29T04:29:30Z

+
+          install -m 0555 "${supervisor_bin}" "${build_context}/openshell-sandbox"
+          cat >"${build_context}/Containerfile" <<'EOF'
+          FROM alpine:3.22


gator-agent

Warning: This post-cache hook resolves mutable alpine:3.22 and mutable APK packages on every fresh test guest. That makes E2E behavior depend on current registry contents and lets an upstream compromise alter the supervisor enforcement environment (CWE-494). Pin the multi-architecture base digest and reproducible package inputs, or consume a trusted digest-qualified supervisor base.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew

gator-agent

PR Review Status

Validation: This maintainer-authored PR remains project-valid as reusable Nix/QEMU test infrastructure for release and E2E work.
Head SHA: 420ddce5e9777a3ac55c25d649852ec56f0f02ae

Review findings:

One blocking QCOW2 validation control-flow defect and two cache safety warnings are inline.
The blocking validator defect can accept prohibited disk structure when Bash suppresses errexit in conditional call contexts.

Durable feedback: Prior trusted maintainer resolutions and waivers remain honored. The findings here are distinct from the resolved checksum, existing-tag validation, and earlier runner/Podman findings.

Docs: Fern docs/navigation are not required for this contributor-facing test infrastructure; nix/test-guest/README.md and architecture/build.md cover the workflow.

Tests: No local tests were run. Please add focused shell coverage for validator failure propagation, the decompression bound, and post-push digest verification.

Checks: Required Branch Checks and Helm Lint are pending mirror validation, but author changes are needed before pipeline watch resumes.

Next state: gator:in-review

drew · 2026-07-29T15:00:37Z

+		((.snapshots? // []) | length == 0) and
+		(.["virtual-size"] > 0) and
+		(.["virtual-size"] <= 68719476736)
+	' <<<"${info}" >/dev/null


gator-agent

Blocking: The structural jq rejection can be masked by the following successful qemu-img check. This function is called through ! and &&, contexts where Bash suppresses errexit for the function body, so disks with backing files, external data files, snapshots, or excessive virtual sizes can be accepted (CWE-20). Make every step explicit: info=$(...) || return 1, jq ... || return 1, and qemu-img check ... || return 1; add rejection tests for each prohibited disk shape.

drew · 2026-07-29T15:00:37Z

+
+	(
+		# Limit the decompressed file to 32 GiB before qemu-img parses it.
+		ulimit -f 67108864


gator-agent

Warning: Bash interprets ulimit -f in 1024-byte units, so this permits a 64 GiB output rather than the documented 32 GiB. A compressed OCI artifact can therefore consume twice the intended host storage before validation (CWE-409). Use 33554432 for 32 GiB and add a mocked decompression-boundary test.

drew · 2026-07-29T15:00:37Z

+			"metadata.json:${TEST_GUEST_CACHE_METADATA_TYPE}" \
+			"disk.qcow2.zst:${TEST_GUEST_CACHE_DISK_TYPE}"
+	)
+	published_digest=$(oras resolve "${remote_ref}")


gator-agent

Warning: Resolving the mutable tag after pushing can race another publisher, causing the printed immutable digest to identify an artifact this process did not upload (CWE-367/CWE-345). Capture the digest directly from the push result, or pull and validate the resolved digest against the local disk checksum before reporting it as trusted.

drew mentioned this pull request Jul 24, 2026

ci(release): run Linux release canaries in QEMU VMs #2472

Draft

10 tasks

drew changed the title ~~test(vm): add composable QEMU test guests~~ test(vm): add reusable QEMU infrastructure for E2E tests Jul 24, 2026

drew mentioned this pull request Jul 24, 2026

test(e2e): add VM-backed E2E suite runner #2473

Open

9 tasks

drew changed the title ~~test(vm): add reusable QEMU infrastructure for E2E tests~~ test(e2e): add reusable QEMU infrastructure for E2E tests Jul 24, 2026

drew force-pushed the drew/e2e-vm-example branch from a9988d0 to 41f2720 Compare July 24, 2026 23:17

drew added 6 commits July 28, 2026 12:28

test(vm): add composable QEMU test guests

9de27e5

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

docs(vm): describe test VM directory structure

2a9d51d

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

refactor(vm): replace shell catalog functions

dc2a466

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

test(vm): add Fedora release guest support

59aa8d9

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

test(vm): enable rootless Podman socket

e2351e2

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

feat(test-guest): add OCI-backed image caching

52cb5e1

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew force-pushed the drew/e2e-vm-example branch from 4f528ef to 52cb5e1 Compare July 28, 2026 19:29

drew marked this pull request as ready for review July 28, 2026 19:47

drew requested review from a team, derekwaynecarr, maxamillion and mrunalp as code owners July 28, 2026 19:47

drew commented Jul 28, 2026

View reviewed changes

Comment thread nix/test-guest/run.sh

Comment thread nix/test-guest/configuration/docker.yml

Comment thread tasks/scripts/package-deb.sh

drew added the gator:in-review Gator is reviewing or awaiting PR review feedback label Jul 28, 2026

perf(test-guest): accelerate cached guest startup

eebc077

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew mentioned this pull request Jul 28, 2026

feat: E2E and release testing improvements #2527

Open

5 tasks

fix(test-guest): address review feedback

49ae902

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew commented Jul 28, 2026

View reviewed changes

Comment thread tasks/scripts/package-deb.sh

Comment thread nix/test-guest/cache.sh Outdated

fix(test-guest): verify OCI cache provenance

9f24004

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew force-pushed the drew/e2e-vm-example branch from 039537a to 9f24004 Compare July 28, 2026 20:42

drew commented Jul 28, 2026

View reviewed changes

Comment thread nix/test-guest/run.sh Outdated

Comment thread nix/test-guest/cache-lib.sh

fix(test-guest): harden cached guest reuse

7f365ef

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew added 2 commits July 28, 2026 14:54

fix(test-guest): refresh runtime setup state

7bd0790

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

feat(test-guest): support E2E runner inputs

418b260

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew commented Jul 28, 2026

View reviewed changes

Comment thread tasks/scripts/package-deb.sh

Comment thread nix/test-guest/run.sh Outdated

drew commented Jul 28, 2026

View reviewed changes

Comment thread nix/test-guest/README.md Outdated

Comment thread tasks/scripts/package-deb.sh

Comment thread nix/test-guest/cache.sh

Comment thread nix/test-guest/run.sh

Comment thread nix/test-guest/run.sh

fix(test-guest): harden runner and OCI reuse

86cf640

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew mentioned this pull request Jul 28, 2026

fix(gator): preserve resolved review feedback #2533

Draft

3 tasks

drew commented Jul 28, 2026

View reviewed changes

drew added gator:watch-pipeline Gator is monitoring PR CI/CD status test:e2e Requires end-to-end coverage and removed gator:in-review Gator is reviewing or awaiting PR review feedback labels Jul 28, 2026

feat(test-guest): prepare Podman E2E artifacts

3d48a73

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew commented Jul 29, 2026

View reviewed changes

drew added gator:in-review Gator is reviewing or awaiting PR review feedback and removed gator:watch-pipeline Gator is monitoring PR CI/CD status labels Jul 29, 2026

elezar mentioned this pull request Jul 29, 2026

feat(podman): relay callbacks through the rootless network namespace #2540

Open

12 tasks

fix(test-guest): address review findings

420ddce

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew commented Jul 29, 2026

View reviewed changes

Uh oh!

Conversation

drew commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Stack

Related Issue

Changes

Testing

Checklist

Uh oh!

copy-pr-bot Bot commented Jul 24, 2026

Uh oh!

copy-pr-bot Bot commented Jul 28, 2026

Uh oh!

drew left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

Uh oh!

Uh oh!

Uh oh!

drew left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

Uh oh!

Uh oh!

drew commented Jul 28, 2026

Uh oh!

drew left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

Uh oh!

Uh oh!

drew left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

Uh oh!

Uh oh!

drew left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

drew left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

drew commented Jul 28, 2026

Uh oh!

github-actions Bot commented Jul 28, 2026

Uh oh!

drew left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

drew Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

drew Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

drew Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

drew left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

drew Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

drew Jul 29, 2026

drew commented Jul 24, 2026 •

edited

Loading